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Abstract 

We  argue  that  the  conventional  privilege  separation  of 
a processor  has  inherent  limitations  in  protecting  soft- 
ware with  higher  security  requirements,  and  hence,  a new 
system  of  protection  should  be  devised  to  overcome  these 
limitations.  To  enable  the  new  protection,  an  operating 
system  needs  to  be  restructured  into  two  layers:  the  secu- 
rity kernel  which  implements  the  new  protection  system, 
and  the  management  kernel  which  manages  resources. 
The  security  kernel  protects  the  applications  even  when 
the  management  kernel  is  compromised.  The  security 
kernel  should  be  made  very  thin  and  simple,  thus  mak- 
ing it  suitable  for  small  devices  like  handsets  and  smart 
sensors  & actuators. 

Limitations  of  Current  Software  Protection 

With  increasing  computation  power  and  storage  capac- 
ity, many  embedded  systems  are  adopting  the  paradigm 
of  user/kernel  separation  of  a processor  [3]  to  provide 
better  software  protection  and  management.  This  protec- 
tion paradigm  is  characterized  by  a complete  separation 
of  privilege.  Code  executing  in  user  mode  (i.e.,  applica- 
tions) is  prevented  from  performing  sensitive  operations, 
whereas  code  executing  in  kernel  mode  (i.e.,  operating 
system)  is  considered  privileged  and  hence,  given  unlim- 
ited power. 

This  simple  protection  mechanism  is  effective  in  pro- 
tecting the  operating  system  (OS),  but  it  does  not  serve 
well  the  security  needs  of  user  applications.  In  an  em- 
bedded environment,  where  applications  usually  perform 
critical  operations  and  carry  sensitive  data,  the  applica- 
tion must  be  protected  as  strongly  as  the  OS.  Although 
the  OS  provides  certain  protection  to  the  applications, 
there  are  inherent  limitations  with  the  simple  user/kernel 
separation  and  complete  reliance  on  the  OS  for  the  appli- 
cation protection. 

First,  there  is  no  effective  second-line  of  defense  that 
applications  can  resort  to  in  case  the  OS  is  compromised. 


Many  applications  need  to  protect  secret/confidential  in- 
formation, but  once  an  attacker  seizes  the  control  of  the 
OS,  it  is  very  easy  for  the  attacker  to  observe/steal  the  in- 
formation. Any  effort  to  further  protect  the  information 
will  be  futile  as  the  attacker  can  exploit  the  OS's  power 
to  subvert,  reverse-engineer,  or  simply  disable  the  pro- 
tection mechanism. 

Second,  verifying  the  correctness  of  an  OS  is  becom- 
ing intractable  as  its  size  and  functionality  continuously 
grow — even  in  an  embedded  environment — to  meet  the 
increasing  demand  for  more  features.  Today's  mobile 
phones,  for  instance,  require  some  features  comparable 
to  those  of  PCs.  Unfortunately,  it  is  difficult  to  reduce  the 
growing  OS  verification  need  due  to  the  coarse-grained 
user/kemel  separation,  where  every  privileged  code  has 
unlimited  power  and  thus,  is  subject  to  verification. 

Third,  the  user/kemel  separation  generates  trust  de- 
pendencies among  software  components,  which  do  not 
generally  correspond  to  the  relations  of  the  component 
providers.  This  mismatch  incurs  assurance  overhead  and 
generates  unwarranted  conflicts  of  interest.  For  example, 
user  applications  must  trust  the  OS.  To  trust  the  OS,  the 
application  providers  need  assurance  of  the  OS’s  trust- 
worthiness. For  the  assurance,  a complete  and  unbiased 
validation  of  the  OS  is  necessary,  but  doing  so  generally 
goes  against  the  interest  of  the  OS  provider  due  to  the 
cost  of  validation  and  the  risk  of  exposing  the  system  to 
others. 

Challenges  in  Designing  New  Application 
Protection 

We  argue  for  the  need  of  another  system  of  protection 
that  can  deal  with  the  above  limitations.  The  new  system 
should  be  able  to  protect  applications  even  in  the  case 
of  OS  compromises,  reduce  the  size  of  code  required  for 
verification,  and  break  the  trust  dependencies  between 
software  components.  To  design  and  implement  such  a 


1 


protection  system,  we  must  overcome  the  following  chal- 
lenges. 

The  first  challenge  is  to  define  an  appropriate  threat 
model  for  user  applications.  We  need  to  identify  the 
security  properties  that  the  applications/users  want  for 
protecting  their  information/data,  so  that  the  new  pro- 
tection mechanism  can  preserve  themselves  even  if  the 
OS  were  compromised.  Unfortunately,  our  problem  is 
not  in  the  secure  communication  domain,  and  thus,  it  is 
difficult  to  borrow  familiar  security  properties  from  that 
domain.  Also,  we  need  to  avoid  over-protection  for  sim- 
plicity. Therefore,  we  need  a threat  model  that  represents 
the  problem  domain  and  captures  essential  security  needs 
of  the  applications. 

The  second  challenge  is  to  preserve  the  OS’s  usual 
management  power.  With  the  new  protection,  however, 
the  OS  is  restricted  somewhat;  the  new  protection  sys- 
tem enforces  certain  rules  and  the  OS  is  prevented  from 
performing  actions  against  the  rules.  However,  the  re- 
striction should  not  obstruct  the  OS  from  performing  a 
legitimate  management  job. 

The  third  challenge  is  to  find  an  implementation  that  is 
small  and  simple  to  verify.  With  the  new  protection,  the 
OS  can  be  verified  less  stringently,  since  applications  can 
still  be  protected  even  when  the  OS  fails  (as  a result  of  its 
compromise).  However,  the  mechanism  that  implements 
the  new  protection  should  be  fully  trusted,  and  hence, 
the  correctness  of  the  implemented  protection  is  critical 
to  the  security  of  the  entire  system. 

Security  Kernel 

The  new  protection  requires  a different  OS  arrangement 
which  consists  of  two  layers  of  kernel:  security  and  man- 
agement kernels.  Running  with  complete  privilege,  the 
security  kernel  is  a very  thin  layer  that  only  implements 
the  new  protection  system.  It  must  be  fully  trusted  and 
must  thus  be  rigorously  verified.  The  management  ker- 
nel, responsible  for  resource  management  and  schedul- 
ing, is  a restricted  version  of  a conventional  OS.  As 
it  runs  on  top  of  the  security  kernel,  applications  are 
still  protected  from  any  compromise  in  the  management 
kernel.  In  this  sense,  it  does  not  have  to  be  trusted 
and  verified.  Both  security  and  management  kernels 
are  protected  from  user  applications  by  the  traditional 
user/kernel  separation. 

Since  it  is  small  and  simple  enough,  the  security  kernel 
can  be  realized  entirely  with  hardware.  Equipped  with  a 
circuitry  that  implements  the  protection  logic,  a proces- 
sor can  extend  its  ISA  to  expose  a programming  interface 
for  the  management  kernel  and  user  applications. 

A software-only  solution  is  also  possible  by  using 
virtualization  techniques.  A virtual  machine  monitor 
(VMM)  is  capable  of  not  only  running  multiple  OSes, 


but  also  realizing  hardware  extensions  or  implementing 
system  services  without  actually  changing  the  real  ma- 
chine [1].  The  VMM  is  more  privileged  than  the  OS  and 
its  perimeter  is  safe.  Therefore,  the  security  kernel  can 
be  implemented  inside  of  the  VMM. 

Although  we  can  implement  the  security  kernel  by 
modifying  a full-fledged  VMM  such  as  Xen  [2],  a full 
VMM  is  not  necessary  as  we  do  not  have  to  run  multi- 
ple OSes.  Instead,  a lightweight  security  kernel  is  pre- 
ferred only  by  using  the  techniques  and  constructs  re- 
quired to  enable  the  hardware  extensions  and  to  safe- 
guard the  VMM's  perimeter. 

Impact  and  Outlook 

The  new  software  protection  system  will  make  long- 
term impacts  since  it  relaxes  many  assumptions  currently 
made  when  software  systems  are  composed.  For  in- 
stance, the  management  OS  is  no  longer  assumed  to  be 
trusted,  thus  creating  opportunities  for  design  of  ambi- 
tious distributed  systems  which  were  risky  under  the  as- 
sumption of  trusted  OS.  Also,  existing  software  systems 
can  be  retroactively  redesigned  to  exploit  the  enlarged 
design  space,  thus  making  them  more  reliable  with  min- 
imal additional  effort. 

Conclusion 

The  conventional  user/kemel  separation  is  not  sufficient 
to  meet  the  growing  demand  for  software  protection  in 
embedded  systems.  We  argue  for  the  need  of  a new 
protection  mechanism  that  can  protect  user  applications, 
lessen  verification  overhead,  and  break  trust  dependen- 
cies. The  new  protection  is  enforced  by  a ‘security  ker- 
nel' which  can  be  realized  as  a lightweight  software  layer 
using  virtualization  techniques,  making  it  suitable  for 
small  devices  and  embedded  systems,  such  as  handsets 
and  smart  sensors  & actuators. 
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Background 

# Why  application  software  protection  in 
distributed  embedded  systems? 

I In  embedded  systems,  application  programs  perform 
mission-critical  tasks  and  carry  sensitive  info 

• Privacy/integrity  of  these  applications  is  critical  to  the 
security  and  robustness  of  any  distributed  embedded 
system 

• Current  approach:  have  the  OS  protect  the 

applications 

I E.g.,  OS  provides  process  isolation,  crypto  services,  etc. 

t Apps  must  trust  OS.  OS  should  be  protected  by 
hardware 

t Processor  protects  OS  via  user/kernel  separation 


My  Position 

Classical  user/kernel  separation  is  too 
coarse  to  conf  idently  protect  app 
software  with  high  security  needs. 
The  limitation  should  be  overcome  by 
creaYmg  a new  protection  system. 
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Classical  User/Kernel  Separation 


Kernel  mode 

Processor  state  with  privilege 

- Execution  mode  for  OS 

- Ability  to  execute  all  instructions 

- Unrestricted  access  to  hardware 


User  mode 


Processor  state  without  privilege 

- Execution  mode  for  applications 

- Can't  execute  system  instructions 

- Restricted  access  to  hardware 


An  autocratic  model  for  separation  of  power:  the  kernel  code 
executes  with  absolute  power 

§»  Entire  security  of  the  system  hinges  on  the  trustworthiness  of  the 
kernel  mode  software  (i.e.,  OS) 

§»  Effective  for  protecting  OS,  but  this  simple  dichotomy  is  too 
coarse  and  there  are  several  limitations 


Limitations  of  user/kernel  Dichotomy 


1.  No  defense  against  OS  compromise 

I There  is  no  effective  2nd  line  of  defense  to  applications 

I Further  protection  of  applications  is  meaningless  as  the 
attacker  can  easily  disarm  any  protection  mechanism 

2.  Difficult  to  reduce  the  OS  verification  overhead 

I Trend:  OS  is  becoming  larger  and  is  from  diverse  sources 

# The  dichotomy  dictates  any  code  that  requires  even 
slightest  privilege  must  execute  in  kernel  mode,  where 
the  code  is  subject  to  complete  verif  ication 

3.  Undue  trust  dependencies 

f App  vendors  require  OS  vendors  not  to  spy  on  the  apps 

• Apps  must  trust  every  component  of  OS 

f Every  OS  component  must  be  validated  (e.g.,  device 
drivers) 


New  Directions  for  Software  Protection 


II  Need  a new  protection  system 

# Protect  applications  even  in  case  of  OS  compromises 

# Lessen  the  kernel  verification  overhead 
i Break  trust  dependencies 

• Challenges  in  designing  such  a protection  system 
t Identifying  an  appropriate  threat  model 

«3  Model  that  captures  essential  security  needs  of  apps 
I Preserving  OS's  management  power 

i3  Restriction  by  the  new  protection  shouldn't  obstruct  OS’s  job 

# Finding  a small  and  simple  enforcing  mechanism 

■3  Implementation  must  be  easily  verifiable 

II  My  proposal:  Separate  security  from  management 

# The  new  protection  system  protects  privacy/integrity  of  apps. 
i It  is  implemented  by  a 'security  kernel'  (continue) 
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Security  Kernel  vs.  Mgmt  Kernel 
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Traditional  layout 
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Hardware 

Security  kernel  approach 


%»  Traditional  OS  Security  kernel  + Management  kernel 
t#  Management  kernel  is  responsible  for  resource  management 
I*  Security  kernel  is  a thin  layer  enforcing  the  new  protection  system 

# It  directly  protects  privacy/integrity  of  applications  data 

• Applications  are  protected  even  if  the  management  kernel  is 
compromised 

U Management  kernel  doesn't  have  to  be  trusted  by  applications 
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Implementation  Alternatives 
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1.  Hardware 

* Processor  is  modified  to  implement  the  protection  logic 

2.  Software:  Using  virtual  machine  monitor  (VMM) 

• A VMM,  sitting  between  HW  and  OS,  can  be  utilized 

# Due  to  size/complexity,  verifying  the  VMM  is  challenging 

3.  Software:  Standalone  security  kernel 

# A thin  layer  implementing  only  the  protection  system 
» Can  be  made  small  and  simple,  thus  easy  to  secure 
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Impact 


U Paradigm  shift  in  designing  secure  distributed 
embedded  systems 

• The  new  application  protection  system  relaxes  many 
assumptions  currently  made 
€ To  ensure  the  security  of  application  software, 
management  OS  no  longer  has  to  be  trusted 
€ It  enables  implementing  ambitious  distributed  systems 
which  were  too  risky  under  the  assumption  of  trusted  OS 


Conclusion 


• Another  system  of  protection  is  needed  to 
overcome  the  limitations  inherent  with  the  coarse 
classical  user/kernel  separation. 

• The  protection  system  must 

t Protect  applications  even  in  case  of  OS  compromises 
it  Lessen  the  kernel  verification  overhead 
I Break  trust  dependencies 
U We  have  been  exploring  approaches  for 
implementing  such  a protection  system 
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